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We investigate the origin of Abelian and non- Abelian type magnetic instabilities induced by Fermi 
surface mismatch between the two pairing fermions in a non-relativistic model. The Abelian type 
instability occurs only in gapless state and the Meissner mass squared becomes divergent at the 
gapless-gapped transition point, while the non- Abelian type instability happens in both gapless and 
gapped states and the divergence vanishes. The non- Abelian type instability can be cured in strong 
coupling region. 
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I. INTRODUCTION 

The fermion Cooper pairing between different species 
with mismatched Fermi surfaces, which was discussed in 
electronic superconductivity many year ago0, 0, El 0], 
promoted new interest in the study of color supercon- 
ducting quark matter in compact stars 0, 0, 0, asym- 
metric nuclear matter with proton-neutron pairing IsL I id. 
and fermionic atom gas with population imbalancejlll 

For color superconducting quark matter with beta 
equilibrium and charge neutrality, the SarmajlJ or 
breached pairing(BP)[5| or gaplessQ state possesses 
paramagnetic response to the external color magnetic 
fields, i.e., the Meissner masses squared of some glu- 
ons are negative 0, 0, 0, 0. Therefore, the homo- 
geneous and isotropic gapless state is dynamically un- 
stable in weak coupling BCS region, and some spatially 
inhomogeneous and anisotropic states are energetically 
favoredjll S El El . 

Since the 8th gluon corresponds to the diagonal gen- 
erator T$ in color space, and in an Abelian system the 
Meissner mass squared is proportional to the superfluid 
density, the chromomagnetic instability of the 8th gluon 
in the gapless two flavor color superconductivity and the 
negative superfluid density in the BP superfluidity are 
of the same type instabilities pil Eol l2q . Generally, 
the Abelian type magnetic instability is related to the 
Fermi surface topology [26j. In weak coupling region, one 
branch of fermionic quasiparticle dispersions has two gap- 
less nodes and there are two sharp effective Fermi surfaces 
in momentum space. In this case, the Meissner mass 
squared or the superfluid density is negatively divergent 
at the gapless-gapped transition point where the chemi- 
cal potential mismatch between the two pairing fermions 
is equal to the pairing gap. At strong enough coupling, 
the branch of the gapless fermionic excitation has only 
one gapless node, and therefore there is only one sharp 



effective Fermi surface in momentum space. This type 
of gapless state is free from both Sarma and magnetic 
instabilities }^ I27I Eel and is energetically more favored 
than the mixed phase |29( and LOFF phase0,0. 

There exists another type of magnetic instability for 
the 4- 7th gluons[lJ| in the two flavor color superconduc- 
tivity. We call it non- Abelian magnetic instability. Very 
different from the Abelian instability for the 8th gluon, 
the Meissner mass squared for the 4- 7th gluons is neg- 
ative in both the gapless and gapped states and is not 
divergent at the gapless-gapped transition point. It is 
recently argued that the LOFF state suffers also nega- 
tive Meissner masses squared for the 4- 7th gluons and 
the gluonic phase will be energetically favored in a wide 
range of coupling [22I l30l |. 

In this paper, we will show that the non-Abelian in- 
stability is related only to the non-Abelian gauge sym- 
metry and is independent of the details of the attractive 
interaction between the two pairing fermions. Any sys- 
tem which possesses a SU(3) gauge symmetry may suf- 
fer this type of magnetic instability. By regarding the 
gauge field as the pseudo Nambu-Goldstone current po|. 
the non-Abelian instability analysis can be applied to 
the systems of condensed matter or fermionic atom gas 
where there is no real non-Abelian gauge field. The pa- 
per is organized as follows. We present the general treat- 
ment for a non-relativistic fermion system in Section ITT1 
then investigate the magnetic instabilities in Abelian and 
non-Abelian cases in Sections II I II and IIVI respectively, 
and summarize in Section We use the natural unit of 
c = h = ks = 1 through the paper. 



II. GENERAL FRAME 

For a non-relativistic fermion system, the partition 
function of the system can be generally expressed as 



Z = 



[d^][di,]e S ^{- d ^ + ^ +Ci ^ (1) 
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in the imaginary time formalism of finite temperature 
field theory with coordinates = (r = it, x), where 
J = J d 3 x. Jq dr is the space-time integration with 
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(3 = 1/T related to the temperature T of the system, ijj, m 
and ll are, respectively, the fermion field, mass and chem- 
ical potential, and the interaction sector £ int provides an 
attractive coupling between the pairing fermions. 

Since a fermion can possess possible inner degrees of 
freedom, the quantities ip, m and /i are normally matri- 
ces in inner space. To simplify the calculation, we neglect 
in this paper mass difference and consider only chemical 
potential difference in the inner space. Generally, due 
to the introduction of inner degrees of freedom, the sys- 
tem may have some gauge symmetry, such as a SU(3) 
gauge symmetry which can, for instance, be realized in 
a 40 K atom gas with three hyperfine statesj^. Suppose 
the system under consideration has a gauge symmetry 
with N generators T a (a = 1,2, ...,N), we can introduce 
a model gauge field A° (a = 1,2,..., N) corresponding to 
the gauge symmetry, and the partition function including 
the gauge field can then be written as 

Z = J [d^\m[dA]e SA e 5 ^^ DT+ ^ + ^' +C ^ , (2) 

where the gauge field enters the theory via the standard 
gauge coupling 

D, t = 9 M - igT a A^ (3) 

with the gauge coupling constant g, and Sa is the action 
for the gauge field sector which is not shown explicitly 
here. 

Due to the attractive interaction between the fermions, 
the BCS pairing occurs and breaks the gauge symmetry. 
The order parameter field 4> which describes the spon- 
taneous symmetry breaking is generally a matrix in the 
inner space. For A = (0) ^ 0, the symmetry group of 
the Lagrangian is spontaneously broken down to some 
subgroups, and some components of the gauge field cor- 
responding to the broken generators will obtain the so- 
called Meissner mass via Higgs mechanism. 

What we are interested in in this paper is how the 
Fermi surface mismatch between the pairing fermions 
affects the behavior of the Meissner mass, especially, 
whether it is real and physical. When some Meissner 
masses squared become negative, the system suffers mag- 
netic instability, and the state with gauge field conden- 
sation (A a ) which breaks the rotational symmetry 
is energetically more favored than the homogeneous and 
isotropic state. 

The model gauge field A a ^ we introduced above is in 
fact not necessary for the magnetic instability analysis. 
If their is no such a gauge field in the theory, for in- 
stance in a fermionic atom gas, it can be replaced by 
the space-time derivative of the phase fluctuation or the 
Nambu-Goldstone current [2(j corresponding to the bro- 
ken generators. Generally, we can define a local phase 
transformation by 

V(x) =e i ^-=^^ Ta , (4) 

where <p a {x) can be regarded as the phase field related 
to the generator T a . Applying the above transformation 



to the fermion field, 

1>(x)=V(x) X (x), ip\x)=x t (x)Vi(x), (5) 

we derive the partition function of the system in terms 
of the fields x an d <fi, 

z = J [d X *]M[M^(~ VT+ £ + ^ x+Cint ] . (6) 

where the derivative T>^ is defined as 

2?„ =d„- iV„(x) (7) 

with the Nambu-Goldstone current 

V fi (x)=V*(x)(id fl )V(x). (8) 

To the linear order in the phase field ip, the Nambu- 
Goldstone current takes the form 

N 

V^x) =J2 d ^a(x)T a . (9) 

a=l 

From the comparison with the standard gauge coupling 
(J3J), the model gauge field A® and the nonzero gauge 
field condensation (A a ) ^= in symmetry breaking state 
can be replaced, respectively, by the Nambu-Goldstone 
current d^fa and its ensemble average poj. 

(V^a) o- (io) 

Therefore, even for those systems without real gauge 
field, the calculation of the Meissner mass for the model 
gauge field is still meaningful. 

The gauge field condensation or spontaneously gener- 
ated Nambu-Goldstone current can be absorbed into a 
LOFF state where the order parameter takes the form 

(<Kx)) = e- 2iT ^A, (11) 

if we identify the wave vector q a = <?(A a ) = (Vip a ). 

The thermodynamic potential Q of the system is a 
function of the order parameter A and pair momentum 
q a , and the physical values of A and q a at fixed temper- 
ature and chemical potentials in the ground state corre- 
spond to the minimum of the thermodynamic potential, 

M = < M^ >0 ' ^ = ' W a >0 - (12) 

Since fi(A, q ) in the LOFF state can be expanded in 
powers of q a at the vicinity of q a = 0, 

n(A, q a ) = fi(A, 0) + ^M 2 b q a ■ q b + (13) 

the coefficients M 2 b are just the Meissner masses squared 
matrix for the gauge field. If some of its elements are 
negative, the LOFF state is energetically favored. For 
an Abelian system, T a — 1, such a LOFF state is just 
the classical single- wave LOFF stated 0, and for a non- 
Abelian system, it can be regarded as an extended non- 
Abelian LOFF state HHH. 
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III. ABELIAN GAUGE SYMMETRY 

For an Abelian system with two species of fermions 
gauge field can be introduced 
into the system via the covariant derivative 



where ~^2 k — T J2 n J m^s means fermion frequency sum- 
mation and vector momentum integration in the momen- 
tum space fc M = (fco = iu> n , k) with w„ = 2wkT for bosons 
and Lo n = (2n + l)nT for fermions. The polarization ten- 
sor Tlij (i,j = 1, 2, 3) can be explicitly expressed as 



D„ 



<9 M - ieA,j, 



(14) 



with gauge coupling e, and the attractive interaction be- 
tween the two species can be modeled by a point inter- 
action with coupling constant G, 



(15) 



where we have assumed that the pairing can occur only 
between different species of fermions, indicated by the 
antisymmetric tensor {s) a f3 = £q/3 in the inner space. The 
order parameter describing spontaneous U(l) symmetry 
breaking is defined as the expectation value of the pair 
field, 



A = -2G{ilFieip). 



(16) 



In a homogeneous and isotropic superconductor, A can 
be set to be real. 

To see clearly the effect of the Fermi surface mismatch 
on the Meissner mass, we replace the individual chemical 
potentials \i\ and fi2 in /i = diag(fii, /12) by the average 
chemical potential fi = (fi\ + ^2) /2 and the chemical 
potential difference Sfi = (/ii — 112) /2 and calculate the 
Meissner mass of the gauge field as a function of the 
pairing gap A and the mismatch Sfi at fixed /2. 

The way to calculate the Meissner mass is quite stan- 
dard. We firstly integrate out the fermionic degrees 
of freedom and then obtain the effective action for the 
gauge field. With the Nambu-Gorkov field defined as 
\& = (V>, V^) T \ the effective action can be expressed as 



Seff [A] =S A + ^ -I Trln [S' 1 + A] , 



where S 1 is the inverse of the fermion propagator in 
Nambu-Gorkov space 



s- 1 



iAs 



iAe 
-d T - vi _ 

' 2m h 



(18) 



and A is a matrix related to the gauge field, A 
diag (A + , A~) with the elements A^ defined as 



A ± = ±eA° =f ^— A 2 — - — (V • A + A • V). (19) 
2m 2m 



(21) 



r 

^-k^ 3 - k] ) TY{s{p)S{p _ k)) 



with the matrix £ defined as £ = diag (I2, where I2 
is the two dimensional identity matrix in the inner space. 
The Meissner mass of the gauge field is defined as 



fci%/|k| 2 )n^(o,k). 



(22) 



Note that the polarization tensor is composed of two 
parts: The first term is the diamagnetic term which is re- 
lated to the fermion number density and the second term 
is the paramagnetic term including the one-loop diagram 
of the fermion propagator. This behavior is quite differ- 
ent from that of a relativistic system|3| where the one- 
loop diagram contributes to both diamagnetic and para- 
magnetic terms. Another convenient way to calculate the 
Meissner mass is through the potential curvature [l8l l25j . 



1 3 

3 .4^ 



d 2 S eff [A] 



dAtdAj 



(23) 



A=0 



Obviously, the two methods are equivalent 
Defining the notation ea = 



A 2 with e p — 



p /(2m) — fi, we can explicitly express the fermion prop- 



(17) agator in the Nambu-Gorkov<g>inner space as 



(Q% 








-&+\ 





g± 










-iT- 


QZ 





\&+ 











S(p) = 



where the nonzero matrix elements are defined as 

iu n ± SfJ, =F e p 



(24) 



(ibJn ± 5fl) 2 

A 

{iu) n ± Sfi) 2 



(25) 



Since we are interested in the Meissner mass which is 
related to the spatial or magnetic component of the gauge 
field, we need only the quadratic term in A of the effective 
action, 

5 e//t A ] = -\y^{-k)U t] {k)A^k), (20) 



The Meissner mass squared m\ can be decomposed 
into a diamagnetic part (m 2 
(m 2 A ) 



A j d and a paramagnetic part 
which come, respectively, from the first and sec- 



ond terms of the polarization tensor, 



m\ 



(26) 
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After a straightforward algebra, they can be evaluated as 



0< 



e 

2m 



1 + G+) e M " 0+ 



Au n O 



- (01 + 0; j e- 

p 



■Gig: 



(27) 



Note that we have added the convergent factors in the 
Matsubara frequency summation in the diamagnetic part 
to ensure the condition m\(A = 0) = 0. 

After the Matsubara frequency summation, we obtain 



e 2 f c 

{ m A) v = 2 / TT 6 ( eA 



p 2 dp 
"2^2" 

OO 4 



1 - ^6 (e A - 5a*) 



■5 M ) 



(28) 



at T = 0. The diamagnetic mass is only related to the 
number density n, 



K) d = — , (29) 
and the paramagnetic mass can be evaluated as 

e 2 SfiQ(Sn -A)p 3 _e (ti-)+p 3 + Q (/J+) 



(m 2 A ) = 

with the definitions 



A 2 



6tt 2 



p± = ^J2mu±, fi±= n± \/5n 2 - A 2 . 
In weak coupling region, we have ft 3> 5fi and 



p\ + p 3 _ (2m/j) 3 / 2 



6vr 2 



3tt 2 



(30) 
(31) 

(32) 



and the Meissner mass squared can be approximately ex- 
pressed as 



ne 
rn 



Sfi@(Sfx - A) 

"TP 



A 2 



(33) 



In the gapless state with <5/i > A, the Meissner mass 
squared is negative, and there is a discontinuity at the 
gapless- gapped transition point 5u — A. This divergence 
and discontinuity come from the delta function in Q2H[). 

When the coupling becomes strong enough, the system 
will enter the BEC region and we have jl < 0. In this 
case there is only one gapless node p+ and the Meissner 
mass squared reads 



m 



ne 

771 



frte(<fc-A)p3e( M+ ) 

^M/J 2 - A 2 67T 2 77 



(34) 



For Jl < 0, p+ becomes much smaller than the Fermi mo- 
mentum pp = (St: 2 !!.) 1 / 3 , the Meissner mass squared in 
the gapless phase can be positive [2^, |23|, and the diver- 
gence at the gapless-gapped transition point disappears 
due to the fact that at Sfj, — A both p + and p_ become 
imaginary. In FigQ] we plot the Meissner mass squared 
in BCS and BEC cases. In BCS case, the gapless state 
suffers magnetic instability, while in the strong coupling 
BEC region, the instability is fully cured. 
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FIG. 1: The Meissner mass squared, scaled by = ne 2 /m, 
as a function of A/S/j,. We take p, — 0.99eF, A = 0.05eF in 
the BCS case and jl = — 0.1eF,A = 0.5eF in the BEC case 
with €f being the Fermi energy for free fermions. 



IV. NON-ABELIAN SYMMETRY 

In this section we consider a system with two types of 
inner degrees of freedom which we call color and flavor. 
They can also be interpreted as spin, isospin and hyper- 
fine state in different systems. Suppose there are two 
flavors and three colors, like two flavor QCD, the system 
can be modelled by the partition function 



Z 



[dipi][di>][dA]e SA x 



(35) 



" - e abc are antisymmetric 



where (e) a p = e a/ 3 and (e c ) a: 
matrices in the flavor and color spaces with the flavor 
index denoted by a, (3 and color by a, b, c. We have in- 
troduced a SU(3) gauge field A a ^ corresponding to the 
color degree of freedom via the derivative in ©. 

We consider pairing only between fermions with differ- 
ent flavors and colors and introduce the order parameter 



-2G(ip T ee 3 iP) 



(36) 



which spontaneously breaks the symmetry from SU(3) 
to SU(2), where we have assumed that only the first two 
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colors participate in the condensate, while the third one 
does not. Similar to the treatment in Abelian case, we 
obtain the effective action 1)17(1 for the gauge field with 
the inverse fermion propagator in Nambu-Gorkov space 



2m 

Aee 3 



(37) 



and the gauge field elements 

A ± =±gT a A° aT ^lAl- 
1m 



2m 



(V-A a +A a -V). (38) 



If the fermion chemical potential is color independent, 
we have /x = diag (/2 + Sfi, jl — Sfj.) (£> I3 with I 3 being 
the three dimensional identity matrix in the color space, 
the Fermi surface mismatch between the pairing fermions 
does not break the SU(3) gauge symmetry explicitly, and 
the quadratic term of the effective action for the magnetic 



component of the gauge field reads 

S™[A] = -i^4(-fc)n«(fc)4(fc) (39) 



with the polarization tensor 



SijSal 



Tr[5(p)S a ] 



(40) 



(2pi - kj)(2pj 
Am 2 



-Tv[S{p)T a S(p - k)T b ] 



where the matrices E a and T a are defined 
as S Q = diag(l 2 ®T 2 ,-I 2 ®T 2 ) and T a = 
diag(h®Ta,h®Ta). 

In the 12-dimensional Nambu-Gorkov<g)flavor<g)color 
space, the fermion propagator can be explicitly expressed 
as 



( G + 
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(41) 



with the elements G± , T± shown in (|25|) and the propa- 
gator Gq± for the unpaired fermions defined as 

$o± = ■ ^ ■ (42) 

Since in the phase with nonzero condensate there 
is still the symmetry SU(2), only the generators 
T4, T5, Tq, T7, Tg are broken, we need to calculate the 
Meissner mass squared only for a = 4,5, 6, 7, 8. 

With the definition for the Meissner mass squared in 
non- Abelian case, 



1 

) ab Q ^ ; ^ 



d 2 S eff [A] 



13 dAfdA b t 

1 J 



(43) 



A=0 



we first consider a = 4, 5, 6, 7. It is straightforward to see 
that all the off-diagonal elements of the matrix ( m \) ab 
in the color subspace vanish and the diagonal elements 
are the same, 



l A) i4 



l A) 55 



K) 



77 



(44) 



Therefore, we need to evaluate m 2 A 4 = (m^) 44 only. 
Taking into account the decomposition, 

m A,4 = ( m AA) d + > ( 45 ) 

the diamagnetic and paramagnetic parts coming, respec- 
tively, from the first and second terms of the polarization 
tensor read 



(m 



9 

8m 



E[(0o 



-(Gt-+So + 

n 2 ~ 2 

— Y 

4m 2 ^ 



- G QA 

GZ 



V 



+ GZ + G+ 
G+) e^°~ 
- Go-Qt 



Gn+G + 



(46) 



Note that only the loops with an unpaired fermion and 
a paired fermion contribute to the paramagnetic term. 
After the Matsubara frequency summations, we obtain 



6 



( m AA)d 
( m A,4)„ 



9 1 


r°° P 2 d P 


2m J 


27T2 [ 


-I. 


/•°° p 4 dp 


m 2 


i 37T2 



0((J/i - e p ) + Q(-6(j, - e p ) + -2-9(8/1 
^(e(^-e p )+e(-«5 M -e p ))- 



1 



1-^ 

J-} 

2e A / 



£A 
A 2 



at T = 0. At weak coupling, the integrals can be ap- 
proximately evaluated and the final result is simplified 

as 



5fl 2 S^y/Sj? 

A 2 A 2 



6 ft 



-Q(Sfx - A) 



where m 2 = ng 2 /m is a A and Sfi independent constant. 

In FigEl we show the Meissner mass squared 
function of the gap parameter A. In the BCS case, both 
the gapless state in the region A /S/j, < 1 and the gapped 
state in the region 1 < A/ 5 ii < y/2 suffer magnetic in- 
stability, and the Meissner mass squared is continuous at 
A/ 6 /j = 1. The continuity at the gapless-gapped transi- 
tion point is totally different from the case with Abelian 
symmetry. We also observed that the magnetic instabil- 
ity can be cured in the strong coupling BEC region. 

Now we turn to the calculation of the Meissner mass 
squared with a — 8. Again we make the diamagnetic and 
paramagnetic decomposition, 



(™a) 88 = (mUi+Kd, 



(48) 



and the paired fermion contributions 



Auj n 



( m ls) 



A 



9 

8m 2 



£ 



p 2 2j 
Y3 



g±G± + GtGt + GIG 



-GZGZ + + 2T+T+ 



(51) 



The terms (m 2 A and (m 2 A 8 ) cancel to each other, 

« 8 )° + « 8 )°-0, (52) 



which means that the unpaired fermions have no contri- 
bution to the Meissner effect corresponding to the 8th 
gauge field, and the final expression of m 2 A 8 is the same 
as that in the Abelian case except for a constant factor, 

,2 



6e 2 



(53) 



Therefore, the magnetic stability analysis for the 8th 
gluon should be the same as the one in the Abelian model. 



and each part can further be divided into a unpaired 
fermion term and a paired fermion term, 



(™ls) d = { m A,s) d + ( m XsY ( 



with the unpaired fermion contributions 



9 

8m 



(G+_+G+ + )e^°- 

Go-G Q 

+GtQt + Q^+ 



8m 2 ^ 3 3 

p 



r ,iu> n 0~ t 



(49) 



0+^0+ 



(50) 
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FIG. 2: The Meissner mass squared m\ A , scaled by m 2 g = 
ng 2 /m, as a function of A/SfX. We take fl = 0.99ef,A = 
0.05eF in the BCS case and p, — — 0.1eF,A = 0.5eF in the 
BEC case. 
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V. SUMMARY 

We have investigated the origin of two types of mag- 
netic instabilities induced by the Fermi surface mismatch 
between the two pairing fermions in a U(l) Abelian 
model and a SU(3) non- Abelian model. Our results show 
that the two types of instabilities are very different. The 
Abelian instability occurs only in the gapless state where 
the chemical potential mismatch <5/i is larger than the gap 
A, and the Meissner mass squared becomes divergent at 
the gapless-gapped transition point <5/i = A. However, 
the non-Abelian instability occurs in both gapless and 
gapped states, and there is no singularity at the gapless- 
gapped transition point. In non-Abelian systems, there 
are both Abelian and non-Abelian magnetic instabilities. 
The former corresponds to the broken diagonal genera- 
tors and the latter to those broken off-diagonal genera- 
tors. While only the paired fermions contribute to the 
Abelian instability, both paired and unpaired fermions 
contribute to the non-Abelian instability which leads 
to the continuity of the Meissner mass squared at the 
gapless-gapped transition point. 

Since the non-relativistic model we considered in this 
paper has the simple but essential pairing structure and 
good ultraviolet behavior, it is quite convenient to study 



the non-Abelian LOFF state [32,|33| and discuss the com- 
petition between the Abelian and non-Abelian LOFF 
states pfflj. 

Our investigation indicates that the magnetic insta- 
bility induced by mismatched fcrmi surfaces does not 
depend on the details of the attractive interaction and 
whether the system is relativistic or non-relativistic. The 
instability is controlled by the symmetry of the sys- 
tem. While a relativistic and a non-relativistic Fermi 
gas may have different dynamics, their magnetic insta- 
bilities are very similar if they have the same symme- 
try For instance, the chromomagnetic instability discov- 
ered in relativistic color superconductor may be closely 
related to the experiments in non-relativistic and non- 
Abelian condensed matter such as the three-component 
Fermi gas, and other new discoveries in the study of 
color superconductivity, such as the abnormal number of 
Nambu-Goldstone bosons 0, 113 an d the instability 
induced by the mismatch between paired and unpaired 
fermions [33, may be realized in condensed matter sys- 
tems. 
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